Math - Stat - 491 - Fall 2014 - Notes - III Hariharan Narayanan

The important ideas related to a Markov chain can be understood by just studying its graph, which has nodes corresponding to states and edges corresponding to nonzero entries in the transition matrix. Figure 1 helps us to summarize key ideas. The first part of this figure shows an irreducible Markov chain on states A,B,C. The graph in this case is strongly connected, i.e., one can move from any node to any other through directed paths. Such a Markov chain has a unique stationary distribution. This Markov chain is also ‘aperiodic’. If you start from any node you can return to it in 2, 3, 4, 5, · · · . steps. So the GCD of all these loop lengths is 1. For such Markov chains if you take a sufficiently large power P of the transition matrix P it will have all entries positive. (In this case however P itself has this property.) If you start from any probability distribution π′ and run an irreducible aperiodic Markov chain for ‘infinite time’ π′TPn will converge to the unique stationary distribution. The value of this distribution will be positive for each state. Next consider the second Markov chain on A′, B′, C ′, D′. Here we can see that from D′ we can reach A′, B′, C ′, but not the other way about. Further if you restrict the Markov chain to A′, B′, C ′ you will get an irreducible chain. The Markov chain on A′, B′, C ′, D′ is not irreducible but has a unique stationary distribution. However it takes zero value on some states. The general rule is the following. If from a given state X you can reach some other state Y but cannot return from Y to X, then the stationary distribution will take value zero on X. We call such states ‘transient’. If you start from any probability distribution π′ and run this Markov chain indefinitely, π′TPn will converge to the unique stationary distribution. The value of this distribution will be positive for each state in R1 but zero for D,D ′.